A Theoretical Analysis of the Repetition Problem in Text Generation

Text generation tasks, including translation, summarization, language models, and etc. see rapid growth during recent years. Despite the remarkable achievements, repetition problem has been observed in nearly all text models undermining performance extensively. To solve problem, many methods have proposed, but there is no existing theoretical analysis to show why this happens how it resolved. In paper, we propose a new framework for problem. We first define Average Repetition Probability (ARP) characterize quantitatively. Then, conduct an extensive of Markov model derive several upper bounds average probability with intuitive understanding. that most are essentially minimizing explicitly or implicitly. Grounded on our theory, is, unfortunately, caused by traits itself. One major reason attributed fact exist too words predicting same word as subsequent high probability. Consequently, easy go back form repetitions dub inflow Furthermore, extend broader deriving concentration bound general model. Finally, based bounds, novel rebalanced encoding approach alleviate thus reducing bound. The experimental results applicable proposed alleviates significantly both translation task modeling task. source code paper can be obtained from https://github.com/fuzihaofzh/repetition-problem-nlg.